System and method for image detection during instrument grasping and stapling

ABSTRACT

Systems and methods for image detection during grasping and stapling include an end effector having a first jaw and a second jaw, an imaging sensor mounted to the end effector and configured to capture images of a material graspable by the end effector, and one or more processors coupled to the end effector and the imaging sensor. The one or more processors are configured to receive the images from the imaging sensor, determine one or more properties of the material based on the images, and display the one or more properties of the material on an interface. In some embodiments, the end effector is part of a medical tool and the material is anatomical tissue. In some embodiments, the end effector further includes one or more of a stapling mechanism configured to staple the material or a cutting mechanism configured to cut the material.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/837,132, filed Apr. 1, 2020, which claims the benefit of U.S.Provisional Application No. 62/828,289, filed Apr. 2, 2019. Each ofthese application is incorporated by reference herein.

TECHNICAL FIELD

The present disclosure relates generally to operation of devices withimaging sensors and end effectors and more particularly to capturingimages of end effectors during grasping, cutting, and/or staplingmaterial.

BACKGROUND

More and more devices are being replaced with computer-assistedelectronic devices. This is especially true in industrial,entertainment, educational, and other settings. As a medical example,the hospitals of today have large arrays of electronic devices inoperating rooms, interventional suites, intensive care wards, emergencyrooms, and/or the like. For example, glass and mercury thermometers arebeing replaced with electronic thermometers, intravenous drip lines nowinclude electronic monitors and flow regulators, and traditionalhand-held surgical and other medical instruments are being replaced bycomputer-assisted medical devices.

These computer-assisted devices are useful for performing operationsand/or procedures on materials, such as the tissue of a patient. Withmany computer-assisted devices, an operator, such as a surgeon and/orother medical personnel, may typically manipulate input devices usingone or more controls on an operator console. As the operator operatesthe various controls at the operator console, the commands are relayedfrom the operator console to a computer-assisted device located in aworkspace where they are used to position and/or actuate one or more endeffectors and/or tools that are mounted (e.g., via repositionable arms)to the computer-assisted device. In this way, the operator is able toperform one or more procedures on material in the workspace using theend effectors and/or tools. Depending upon the desired procedure and/orthe tools in use, the desired procedure may be performed partially orwholly under control of the operator using teleoperation and/or undersemi-autonomous control where the computer-assisted device may perform asequence of operations based on one or more activation actions by theoperator.

Computer-assisted devices, whether actuated manually, teleoperatively,and/or semi-autonomously may be used in a variety of operations and/orprocedures and may have various configurations. Many such instrumentsinclude an end effector mounted at a distal end of a shaft that may bemounted to the distal end of a repositionable or articulated arm. Inmany operational scenarios, the shaft may be configured to be insertedinto the workspace via an opening in the workspace. As a medicalexample, the shaft may be inserted (e.g., laparoscopically,thoracoscopically, and/or the like) through an opening (e.g., a bodywall incision, a natural orifice, and/or the like) to reach a remotesurgical site. In some instruments, an articulating wrist mechanism maybe mounted to the distal end of the instrument's shaft to support theend effector with the articulating wrist providing the ability to alteran orientation of the end effector relative to a longitudinal axis ofthe shaft.

End effectors of different design and/or configuration may be used toperform different tasks, procedures, and functions so as to be allow theoperator to perform any of a variety of procedures on a material.Examples include, but are not limited to, cauterizing, ablating,suturing, cutting, stapling, fusing, sealing, etc., and/or combinationsthereof. Accordingly, end effectors can include a variety of componentsand/or combinations of components to perform these procedures.

In many embodiments, the size of the end effector is typically kept assmall as possible while still allowing it to perform its intended task.One approach to keeping the size of the end effector small is toaccomplish actuation of the end effector through the use of one or moreinputs at a proximal end of the tool, which is typically locatedexternally and/or peripherally to the workspace. Various gears, levers,pulleys, cables, rods, bands, and/or the like, may then be used totransmit actions from the one or more inputs along the shaft of the tooland to actuate the end effector. In some embodiments, a transmissionmechanism at the proximal end of the tool interfaces with variousmotors, solenoids, servos, active actuators, hydraulics, pneumatics,and/or the like provided on a repositionable arm of thecomputer-assisted device. The motors, solenoids, servos, activeactuators, hydraulics, pneumatics, and/or the like typically receivecontrol signals through a master controller and provide input in theform of force and/or torque at the proximal end of the transmissionmechanism, which the various gears, levers, pulleys, cables, rods,bands, and/or the like ultimately transmit to actuate the end effectorat the distal end of the transmission mechanism.

Because of the remote nature of the operation of such end effectors, itmay be difficult in some cases for the operator to directly monitor theend effector and/or grasping of the material. For example, in somecases, other portions of the computer-assisted device, including the endeffector itself, other materials, and/or objects in the workspace may behidden from view during the computer-assisted device's operation. Inother cases it may be difficult to monitor and determine a presence oramount of material available to the end effector to perform a requestedtask (e.g., grasping, sealing, cutting, etc.). Variations in tissueavailability and amount can often result in a grasping, clamping, and/orstapling operation with failed or incorrect operations due to materialvariations over space or time, and different steps of the operation suchas the different stages of forcing staples through the material in astapling operation.

Accordingly, improved methods and systems for the operation ofcomputer-assisted devices, such as computer-assisted devices having endeffectors used to grasp, seal, and/or cut a material are desirable. Insome examples, it may be desirable to image areas proximate to the endeffector during use of the computer-assisted device and the endeffectors so as to help ensure that the tool may be able to successfullyperform a desired procedure on the material.

SUMMARY

Consistent with some embodiments, a computer-assisted device includes anend effector having a first jaw and a second jaw, an imaging sensormounted to the end effector and configured to capture one or more imagesof a material graspable by the end effector, and one or more processorscoupled to the end effector and the imaging sensor. The one or moreprocessors are configured to receive the one or more images from theimaging sensor, determine one or more properties of the material basedon the one or more images, and display the one or more properties of thematerial on an interface.

Consistent with some embodiments, a method performed by one or moreprocessors include operating an end effector having a first jaw and asecond jaw, capturing one or more images of a material grasped by theend effector using an imaging sensor, determining one or more propertiesof the material based on the one or more images, and displayinginformation associated with the one or more properties on an interfacewith the one or more images. The information is associated with anamount of the material that is grasped by the end effector.

Consistent with some embodiments, non-transitory machine-readable mediumcomprising a plurality of machine-readable instructions which whenexecuted by one or more processors are adapted to cause the one or moreprocessors to perform any of the methods described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a computer-assisted system accordingto some embodiments.

FIG. 2 is a simplified diagram showing a tool suitable for use with thecomputer-assisted system of FIG. 1 according to some embodiments.

FIGS. 3A and 3B are simplified side and surface views of an end effectorof a tool having a distal tip mounted imaging sensor according to someembodiments.

FIGS. 4A and 4B are simplified side and surface views of an end effectorof a tool having a side jaw mounted imaging sensor according to someembodiments.

FIG. 5 is a simplified diagram of a method for material manipulationaccording to some embodiments.

FIGS. 6A, 6B, and 6C are simplified orientation view hints for aninterface of a computer-assisted system of FIG. 1 having an end effectorof FIGS. 3A and 3B or FIGS. 4A and 4B according to some embodiments.

FIGS. 7A and 7B are simplified diagrams of the distal end of aninstrument grasping a material according to some embodiments.

In the Figures, elements having the same designations have the same orsimilar functions.

DETAILED DESCRIPTION

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or modules should not be taken aslimiting—the claims define the protected invention. Various mechanical,compositional, structural, electrical, and operational changes may bemade without departing from the spirit and scope of this description andthe claims. In some instances, well-known circuits, structures, ortechniques have not been shown or described in detail in order not toobscure the invention. Like numbers in two or more figures represent thesame or similar elements.

In this description, specific details are set forth describing someembodiments consistent with the present disclosure. Numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments. It will be apparent, however, to one skilled in the artthat some embodiments may be practiced without some or all of thesespecific details. The specific embodiments disclosed herein are meant tobe illustrative but not limiting. One skilled in the art may realizeother elements that, although not specifically described here, arewithin the scope and the spirit of this disclosure. In addition, toavoid unnecessary repetition, one or more features shown and describedin association with one embodiment may be incorporated into otherembodiments unless specifically described otherwise or if the one ormore features would make an embodiment non-functional.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms-such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike-may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations) and orientations (i.e., rotational placements) of theelements or their operation in addition to the position and orientationshown in the figures. For example, if the content of one of the figuresis turned over, elements described as “below” or “beneath” otherelements or features would then be “above” or “over” the other elementsor features. Thus, the exemplary term “below” can encompass bothpositions and orientations of above and below. A device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly. Likewise,descriptions of movement along and around various axes include variousspecial element positions and orientations. In addition, the singularforms “a”, “an”, and “the” are intended to include the plural forms aswell, unless the context indicates otherwise. And, the terms“comprises”, “comprising”, “includes”, and the like specify the presenceof stated features, steps, operations, elements, and/or components butdo not preclude the presence or addition of one or more other features,steps, operations, elements, components, and/or groups. Componentsdescribed as coupled may be electrically or mechanically directlycoupled, or they may be indirectly coupled via one or more intermediatecomponents.

Elements described in detail with reference to one embodiment,implementation, or module may, whenever practical, be included in otherembodiments, implementations, or modules in which they are notspecifically shown or described. For example, if an element is describedin detail with reference to one embodiment and is not described withreference to a second embodiment, the element may nevertheless beclaimed as included in the second embodiment. Thus, to avoid unnecessaryrepetition in the following description, one or more elements shown anddescribed in association with one embodiment, implementation, orapplication may be incorporated into other embodiments, implementations,or aspects unless specifically described otherwise, unless the one ormore elements would make an embodiment or implementation non-functional,or unless two or more of the elements provide conflicting functions.

In some instances, well known methods, procedures, components, andcircuits have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

This disclosure describes various devices, elements, and portions ofcomputer-assisted devices and elements in terms of their state inthree-dimensional space. As used herein, the term “position” refers tothe location of an element or a portion of an element in athree-dimensional space (e.g., three degrees of translational freedomalong Cartesian x-, y-, and z-coordinates). As used herein, the term“orientation” refers to the rotational placement of an element or aportion of an element (three degrees of rotational freedom—e.g., roll,pitch, and yaw). As used herein, the term “shape” refers to a setpositions or orientations measured along an element. As used herein, andfor a device with repositionable arms, the term “proximal” refers to adirection toward the base of the computer-assisted device along itskinematic chain and “distal” refers to a direction away from the basealong the kinematic chain.

Aspects of this disclosure are described in reference tocomputer-assisted systems and devices, which may include systems anddevices that are teleoperated, remote-controlled, autonomous,semiautonomous, robotic, and/or the like. Further, aspects of thisdisclosure are described in terms of an implementation using a surgicalsystem, such as the da Vinci® Surgical System commercialized byIntuitive Surgical, Inc. of Sunnyvale, Calif. Knowledgeable persons willunderstand, however, that inventive aspects disclosed herein may beembodied and implemented in various ways, including robotic and, ifapplicable, non-robotic embodiments and implementations. Implementationson da Vinci® Surgical Systems are merely exemplary and are not to beconsidered as limiting the scope of the inventive aspects disclosedherein. For example, techniques described with reference to surgicalinstruments and surgical methods may be used in other contexts. Thus,the instruments, systems, and methods described herein may be used forhumans, animals, portions of human or animal anatomy, industrialsystems, general robotic, or teleoperational systems. As furtherexamples, the instruments, systems, and methods described herein may beused for non-medical purposes including industrial uses, general roboticuses, sensing or manipulating non-tissue work pieces, cosmeticimprovements, imaging of human or animal anatomy, gathering data fromhuman or animal anatomy, setting up or taking down systems, trainingmedical or non-medical personnel, and/or the like. Additional exampleapplications include use for procedures on tissue removed from human oranimal anatomies (without return to a human or animal anatomy) and forprocedures on human or animal cadavers. Further, these techniques canalso be used for medical treatment or diagnosis procedures that include,or do not include, surgical aspects.

FIG. 1 is a simplified diagram of a computer-assisted system 100according to some embodiments. As shown in FIG. 1 , computer-assistedsystem 100 includes a device 110 with one or more repositionable arms120. Each of the one or more repositionable arms 120 may support one ormore tools 130. In some examples, device 110 may be consistent with acomputer-assisted medical device. The one or more tools 130 may includetools, imaging devices, and/or the like. In some medical examples, thetools may include medical tools, such as clamps, grippers, retractors,cautery tools, suction tools, suturing devices, and/or the like. In somemedical examples, the imaging devices may include endoscopes, cameras,ultrasonic devices, fluoroscopic devices, and/or the like. In someexamples, each of the one or more tools 130 may be inserted into aworkspace (e.g., anatomy of a patient, a veterinary subject, and/or thelike) through a respective cannula mounted to a respective one of theone or more repositionable arms 120. In some examples, a direction ofview of an imaging device may correspond to an insertion axis of theimaging device and/or may be at an angle relative to the insertion axisof the imaging device. In some examples, each of the one or more tools130 may include an end effector that may include an imaging sensor(e.g., a camera or other radiation detection component of the endeffector). The end effector may be capable of capturing an image of ascene at and/or nearby the end effector using the imaging sensor, suchas a scene proximate to a distal tip of the end effector and/or betweena space created by the end effector and/or portions of the end effector.In some examples, the end effector may further be capable of grasping amaterial (e.g., tissue of a patient) located in the workspace prior to,during, and/or after capturing one or more images. The imaging sensormay determine properties of the material prior to, during, and/or aftergrasping the material, which may be used to determine a percentage ofthe end effector filled with the material prior to, during, and/or afterthe grasping the material. In some embodiments, computer-assisted system100 may be found in an operating room and/or an interventional suite.

Device 110 is coupled to a control unit 140 via an interface. Theinterface may include one or more cables, connectors, and/or buses andmay further include one or more networks with one or more networkswitching and/or routing devices. Control unit 140 includes a processor150 coupled to memory 160. Operation of control unit 140 is controlledby processor 150. And although control unit 140 is shown with only oneprocessor 150, it is understood that processor 150 may be representativeof one or more central processing units, multi-core processors,microprocessors, microcontrollers, digital signal processors, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), graphics processing units (GPUs), tensor processingunits (TPUs), and/or the like in control unit 140. Control unit 140 maybe implemented as a stand-alone subsystem and/or as a board added to acomputing device or as a virtual machine.

Memory 160 may be used to store software executed by control unit 140and/or one or more data structures used during operation of control unit140. Memory 160 may include one or more types of machine readable media.Some common forms of machine readable media may include floppy disk,flexible disk, hard disk, magnetic tape, any other magnetic medium,CD-ROM, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM,any other memory chip or cartridge, and/or any other medium from which aprocessor or computer is adapted to read.

As shown, memory 160 includes a grasp control module 170, an imagecontrol module 180, and one or more images 190, that may be used tocontrol, monitor, and/or provide properties associated with one of theone or more of tools 130 as is described in further detail below. Andalthough FIG. 1 shows grasp control module 170, image control module180, and the one or more images 190 as separate elements stored within asame memory 160 of a same control unit 140, other configurations arepossible. In some examples, grasp control module 170, image controlmodule 180, and the one or more images 190 may be combined partiallyand/or completely within a same module. In some examples, grasp controlmodule 170, image control module 180, and the one or more images 190 mayalternatively be stored in different memories, and/or associated withdifferent control units. Further, even though grasp control module 170,image control module 180, and the one or more images 190 arecharacterized as software modules, each may be implemented usingsoftware, hardware, and/or a combination of hardware and software.

In some embodiments, grasp control module 170 is responsible formanaging the mechanical operation of the one or more tools 130. In someexamples, grasp control module 170 may monitor one or more sensors usedto track the position, orientation, articulation, and/or mechanicalactuation of the one or more tools 130 and their respective endeffectors and/or one or more material properties of material beinginteracted with by the one or more tools 130 and their respective endeffectors. In some examples, grasp control module 170 may control theposition, orientation, articulation, and/or mechanical actuation of theone or more tools 130 and their respective end effectors using one ormore actuators based on the monitoring and/or the one or more images190. In some examples, control of the position, orientation,articulation, and/or mechanical actuation of the one or more tools 130and their respective end effectors may include controlling one or moredegrees of freedom including, as examples, an insertion depth, a roll, apitch, a yaw, a wrist articulation, an angle between jaws, a force ortorque applied, and/or the like.

In some embodiments, image control module 180 is responsible formanaging an imaging sensor, device, and/or camera attached to orassociated with of the one or more tools 130. In some examples, imagecontrol module 180 may monitor one or more imaging sensors used tocapture images of scenes at and/or proximate to the one or more of tools130 and/or track a location of the one or more tools 130 and theirrespective end effectors. In some examples, image control module 180 maydetermine one or more material properties of materials being interactedwith by the one or more tools 130 and/or their respective end effectors.In some examples, image control module 180 may control the energydelivered by an illuminating source, such as an infrared and/or visiblelight source, associated with the one or more imaging sensors using oneor more light sources, illumination systems, signal generators, opticalfibers, and/or the like, which may be utilized during the use of theimaging sensor to capture the one or more images 190.

In some embodiments, the one or more images 190 include visual imagedata and accompanying properties determined using the visual image dataduring the use of grasp control module 170 and/or image control module180 to control mechanical and/or energy movement and delivery,respectively, of the one or more tools 130 and their respective endeffectors. In some examples, the one or more images 190 may include oneor more still images, videos (e.g., a collection of still images in asequence), and/or the like, which may be captured by the imaging sensorattached and/or associated with the one or more tools 130 and/or theirrespective end effectors as is described in further detail below. Insome examples, one or more properties may be determined from the one ormore images, such as properties of the material grasped and/or graspableby the one or more tools 130 and/or their respective end effectors. Insome examples, the one or more properties may include a desiccation ofthe material, a presence or absence of the material grasped and/orgraspable by the one or more tools 130 and/or their respective endeffectors, a pressure applied to the material, a transmissivity of thematerial and/or moisture within and/or in contact with the material, afluorescence of the material, a length of fill of a space between theone or more tools 130 and/or their respective end effectors, and/or thelike, and may include a change in the aforementioned properties.

In some examples, the one or more properties of the one or more images190 may be used to determine and/or estimate further data within the oneor more images 190, including a percentage fill of a grasping mechanismof the one or more tools 130 and/or their respective end effectors. Insome embodiments, image control module 180 may also be responsible forproviding orientation hints and/or visual clues that may be outputthrough an interface that informs an operator of an orientation of theone or more tools 130 and/or their respective end effectors within aworkspace using the one or more images 190 and/or their correspondingproperties. For example, image control module 180 may provide an image,representation, and/or indication of a view of the one or more tools 130and/or their respective end effectors within a workspace, which may beused with, prior to, and/or overlaid on the one or more images toprovide the orientation hints. In some examples, image control module180 may orient the view of an imaging sensor attached to and/orassociated with the one or more tools 130 and their respective endeffectors relative to one or more axes of the workspace. In someembodiments, image control module 180 may provide an animation of theorientation hints prior to, during, and/or after display of the one ormore images 190, such as an animation of a T-pose of a body in a proneposition that is animated to display entrance of the one or more tools130 and/or their respective end effectors to a workspace (e.g., the bodyof a patient).

In some embodiments, image control module 180 may also provide videostabilization to the one or more images constituting a video during useand/or movement of the one or more tools 130 and their respective endeffectors. For example, image control module 180 may fix relative pointswithin a workspace to relative locations within an interface during theuse and/or movement of the one or more tools 130 and/or their respectiveend effectors. Thus, image control module 180 may apply one or moreimage/video stabilization algorithms utilizing the fixed points toadjust for the use and/or movement of the one or more tools 130 and/ortheir respective end effectors and/or other destabilizing events fromthe workspace. For example, fixed points in tissue of a patient may berelatively fixed to locations within an interface to provide for suchimage/video stabilization during use and/or movement of the one or moretools 130 and/or their respective end effectors.

As discussed above and further emphasized here, FIG. 1 is merely anexample which should not unduly limit the scope of the claims. One ofordinary skill in the art would recognize many variations, alternatives,and modifications. According to some embodiments, computer-assistedsystem 100 may include any number of computer-assisted devices witharticulated arms and/or instruments of similar and/or different indesign from computer-assisted device 110. In some examples, each of thecomputer-assisted devices may include fewer or more articulated armsand/or instruments.

According to some embodiments, the arrangement of grasp control module170, image control module 180, and/or the one or more images 190 may bedifferent than as depicted in FIG. 1 . In some examples, grasp controlmodule 170, image control module 180, and/or the one or more images 190may be distributed across more than one control unit. In some examples,grasp control module 170 and image control module 180 may be included ina single control module. In some examples, the one or more images 190may be included in grasp control module 170 and/or image control module180.

FIG. 2 is a simplified diagram showing a tool 200 suitable for use withthe computer-assisted system 100 according to some embodiments. In someembodiments, tool 200 may be consistent with any of the tools 130 ofFIG. 1 . The directions “proximal” and “distal” as depicted in FIG. 2and as used herein help describe the relative orientation and locationof components of tool 200.

As shown in FIG. 2 , tool 200 includes a long shaft 210 used to couplean end effector 220 located at a distal end of shaft 210 to where thetool 200 is mounted to a repositionable arm and/or a computer-assisteddevice at a proximal end of shaft 210. Depending upon the particularprocedure for which the tool 200 is being used, shaft 210 may beinserted through an opening (e.g., a body wall incision, a naturalorifice, and/or the like) in order to place end effector 220 inproximity to a workspace, such as a remote surgical site located withinthe anatomy of a patient. As further shown in FIG. 2 , end effector 220is generally consistent with a two-jawed gripper-style end effector,which in some embodiments may further include an imaging sensor and/orimage capture mechanism as is described in further detail below withrespect to FIGS. 3A, 3B, 4A, 4B, 5, 6A, 6B, and/or 6C. However, one ofordinary skill would understand that different tools 200 with differentend effectors 220 are possible and may be consistent with theembodiments of tool 200 as described elsewhere herein.

A tool, such as tool 200 with end effector 220 typically relies onmultiple degrees of freedom (DOFs) during its operation. Depending uponthe configuration of tool 200 and the repositionable arm and/orcomputer-assisted device to which it is mounted, various DOFs that maybe used to position, orient, and/or operate end effector 220 arepossible. In some examples, shaft 210 may be inserted in a distaldirection and/or retreated in a proximal direction to provide aninsertion DOF that may be used to control how deep within the workspaceend effector 220 is placed. In some examples, shaft 210 may be ablerotate about its longitudinal axis to provide a roll DOF that may beused to rotate end effector 220. In some examples, additionalflexibility in the position and/or orientation of end effector 220 maybe provided by an articulated wrist 230 that is used to couple endeffector 220 to the distal end of shaft 210. In some examples,articulated wrist 230 may include one or more rotational joints, such asone or more roll, pitch or yaw joints that may provide one or more“roll,” “pitch,” and “yaw” DOF(s), respectively, that may be used tocontrol an orientation of end effector 220 relative to the longitudinalaxis of shaft 210. In some examples, the one or more rotational jointsmay include a pitch and a yaw joint; a roll, a pitch, and a yaw joint, aroll, a pitch, and a roll joint; and/or the like. In some examples, endeffector 220 may further include a grip DOF used to control the openingand closing of the jaws of end effector 220. Depending upon theconfiguration, end effector 220 may include two moveable jaws that arearticulated with respect to each other about a hinge point located neara proximal end of end effector 220 or one fixed jaw and one moveable jawthat is articulated with respect to the fixed jaw about the hinge point.End effector 220 may also include an activation DOF used to control theextension, retraction, and/or operation of a stapling and cuttingmechanism as is described in further detail below.

Tool 200 further includes a drive system 240 located at the proximal endof shaft 210. Drive system 240 includes one or more components forintroducing forces and/or torques to tool 200 that may be used tomanipulate the various DOFs supported by tool 200. In some examples,drive system 240 may include one or more motors, solenoids, servos,active actuators, hydraulic actuators, pneumatic actuators, and/or thelike that are operated based on signals received from a control unit,such as control unit 140 of FIG. 1 . In some examples, the signals mayinclude one or more currents, voltages, pulse-width modulated waveforms, and/or the like. In some examples, drive system 240 may includeone or more shafts, gears, pulleys, rods, bands, and/or the like whichmay be coupled to corresponding motors, solenoids, servos, activeactuators, hydraulics, pneumatics, and/or the like that are part of thearticulated arm, such as any of the repositionable arms 120, to whichtool 200 is mounted. In some examples, the one or more drive inputs,such as shafts, gears, pulleys, rods, bands, and/or the like, may beused to receive forces and/or torques from the motors, solenoids,servos, active actuators, hydraulics, pneumatics, and/or the like andapply those forces and/or torques to adjust the various DOFs of tool200.

In some embodiments, the forces and/or torques generated by and/orreceived by drive system 240 may be transferred from drive system 240and along shaft 210 to the various joints and/or elements of tool 200located distal to drive system 240 using one or more drive mechanisms250. In some examples, the one or more drive mechanisms may include oneor more gears, levers, pulleys, cables, rods, bands, and/or the like. Insome examples, shaft 210 is hollow and drive mechanisms 250 pass alongthe inside of shaft 210 from drive system 240 to the corresponding DOFin end effector 220 and/or articulated wrist 230. In some examples, eachof drive mechanisms 250 may be a cable disposed inside a hollow sheathor lumen in a Bowden cable like configuration. In some examples, thecable and/or the inside of the lumen may be coated with a low-frictioncoating such as polytetrafluoroethylene (PTFE) and/or the like. In someexamples, as the proximal end of each of the cables is pulled and/orpushed inside drive system 240, such as by wrapping and/or unwrappingthe cable about a capstan or shaft, the distal end of the cable movesaccordingly and applies a suitable force and/or torque to adjust one ofthe DOFs of end effector 220, articulated wrist 230, and/or tool 200. Insome examples, drive system 240 may be controlled and/or receiveinstructions from a grasp control module, such as grasp control module170.

In some embodiments, tool 200 further includes an imaging system 260located at the proximal end of shaft 210. Imaging system 260 includesone or more components for capturing one or more images, includingvideo, using an imaging sensor attached to and/or associated with tool200. In some examples, the imaging sensor may be capable of capturingvisual images and/or video of a scene at and/or proximate to the endeffector 220 located at the distal end of shaft 210. In someembodiments, imaging system 260 may also and/or alternatively be capableof capturing other types of signals and/or radiation includingultrasonic, radio frequency, electrical, magnetic, thermal, light,and/or the like. In some examples, imaging system 260 may include one ormore illumination systems that may apply illumination to a scene, suchas visible light (e.g., white and/or colored light), infraredlight/radiation, and/or the like. In some embodiments, imaging system260 may also include components used by an image control module, such asimage control module 180, to determine one or more properties of amaterial from the one or more images and accompanying data (e.g., apercentage fill of a two-jawed gripper-style end effector with materialand/or orientation hints for an orientation of tool 200 in a workspace),as well as provide additional features (e.g., fixed point detection on amaterial for image/video stabilization).

In some embodiments, the one or more images captured by an imagingsensor located at end effector 220 and distal to imaging system 260 maybe transferred from the imaging sensor and along shaft 210 to imagingsystem 260 using one or more image delivery mechanisms 270. In someexamples, the one or more image delivery mechanisms 270 may include oneor more wires, cables, optical fibers, and/or like. In some examples,shaft 210 is hollow and image delivery mechanisms 270 pass along theinside of shaft 210 from imaging system 260 to end effector 220. Imagedelivery mechanism 270 therefore assist in data exchange when capturingone or more images of a scene of a material within the workspace byimaging system 260 using the distally located imaging sensor at endeffector 220. In some examples, imaging system 260 may be controlledand/or receive instructions from an image control module, such as imagecontrol module 180.

FIGS. 3A and 3B are simplified side and top views of an end effector 300of a tool according to some embodiments. In some embodiments, endeffector 300 is consistent with either or both of the jaws of endeffector 220. As shown in FIGS. 3A and 3B, end effector 300 is locatedat the distal end of tool 200 (e.g., end effector 220) and includes amechanism for jaw closure, tissue stapling, and tissue cutting. Andalthough end effector 220 is shown and described with one fixed and onemovable jaw, one of ordinary skill would understand that the distal endof tool 200 could be modified to use two movable jaws. It should befurther understood that although the description below is in the contextof a grasping, stapling, and cutting instrument that simultaneouslygrasps, staples, and cuts tissue, the aspects so described may beapplicable to instruments with or without cutting features, instrumentssupporting fusing rather than stapling, and/or the like.

As shown in FIG. 3A, end effector 300 shows a cut-way side view of endeffector 220 prior to actuation so that the jaws of end effector 220 areshown in an open position. As shown, end effector includes a first jaw340 that is generally fixed. As further shown in FIG. 3A, end effector300 of end effector 220 further includes a second jaw 310 that ismovable about a pivot point 330 near its proximal end. In the context ofa stapling instrument, second jaw 310 may alternatively be referred toas anvil 310. In some embodiments, pivot point 330 is configured so thatupon initial actuation of end effector 220, a gap between first jaw 340and second jaw 310 is rapidly reduced until a material is graspedbetween first jaw 340 and second jaw 310. Actuation of end effector 220is accomplished by grasp control module 170 with drive system 240 fromthe proximal end of end effector 220 providing mechanical actuation,electrical signals, and/or the like to the distal end of end effector220. Thus, one or more components of end effector 300 are coupled to oneor more of the drive mechanisms 250. First jaw 340 further includes afirst jaw surface 350 and second jaw 310 further includes a second jawsurface 320. In some examples, second jaw surface 320 is generallyplanar and is parallel with first jaw surface 350 of first jaw 340 whenend effector 300 is closed relative to the opposing jaws. Thus, secondjaw surface 320 is at an angle to first jaw surface 350 when endeffector 300 is open. As shown, end effector 300 may be utilized forgrasping, cutting, and/or stapling effects when pivoted around pivotpoint 330.

In FIG. 3A, end effector 300 further includes an imaging sensor 360 thatmay capture images of a scene at and/or in proximity to end effector 300during use, movement, and/or actuation of end effector 300. Imagingsensor 360 may correspond to a sensor capable of capturing still imagesand/or video of a scene, such as a sensor array having one or moresensor pixels arranged in array or other fashion that is capable ofdetecting incident radiation on the sensor pixel. In some embodiments,imaging sensor 360 may be capable of capturing visual images at and/ornearby tool 200 using visible light. In such embodiments, imaging sensor360 may correspond to a charge-coupled device (CCD) and/or complementarymetal-oxide semiconductor (CMOS) image sensor. Imaging sensor 360 maycorrespond to a rectangular imaging sensors (e.g., a 400×400 array),which may include a regular or wide angle lens, able to capture an imageat a point in time based on incoming radiation. Imaging sensor 360 mayalso correspond to a single line of sensors that may reconstruct animage over time based on movement of end effector 220. However infurther embodiments, imaging sensor 360 may also and/or alternativelyinclude a thermal and/or infrared sensor that may be capable ofcapturing infrared images of the scene. Imaging using imaging sensor 360on end effector 220 is accomplished by image control module 180 withimaging system 260 from the proximal end of end effector 220 exchangingelectrical signals and the like with imaging sensor 360. Thus, one ormore components of imaging sensor 360 are coupled to the one or more ofthe image delivery mechanisms 270. It should be further understood thatalthough the description below is in the context of an image sensorinstrument that captures visual images and/or infrared images, theaspects so described may be applicable to instruments with other typesof sensors capable of capturing other types of radiation and/or thelike.

In some embodiments, imaging sensor 360 and/or another component of endeffector 300 may include an illumination or lighting system (not shown),which may provide illumination to the scene to be captured using imagingsensor 360. In such embodiments, an emitter may be located with, at,and/or proximate to imaging sensor 360 to provide the illumination, andmay correspond to a light emitting diode (LED), light pipe, luminescentand/or leaky optical fiber that emits light, and/or the like. Theillumination may correspond to visible light, such as colored or whitelight, which may be used to illuminate the scene during capture ofvisual images. The illumination may also correspond to non-visiblelight, such as infrared light and/or the like, which may providefluorescence of a scene, infrared imaging, and/or the like. Theillumination system may further include a transmission system throughtool 200 (e.g., from one or more modules/systems located at the proximalend of tool 200 to the distal end of tool 200), which may beincorporated with the one or more of the image delivery mechanisms 270.The transmission system may include one or more optical fibers thattransmit the illuminating light to end effector 220 from a light sourceat the proximal end of tool 200. In some embodiments, the light sourcemay correspond to a light engine having red, green, and blue LEDs and/orlaser diodes that may mix to create colored/white light, a white LED, aninfrared light source, an ultraviolet light source, and/or the like. Insome examples, an imaging system, such as imaging system 260 (and/or inconjunction with image control module 180) may be used to control theillumination amount, cycles, and/or limits applied to the illuminationsystem. For example, in order to reduce and/or avoid incident heat on ascene, the imaging system may monitor, control, and/or reduceillumination as necessary to avoid unwanted heating of the scene. Insome embodiments, the illumination emitter may be located on one or moreof first jaw 340 and/or second jaw 310, or may be located elsewhere,such as on an endoscope associated with end effector 300 and locatedproximate to imaging sensor 360 so that illumination may be provided tothe scene.

In FIG. 3B a second orientation of end effector 300 is shown, wheresecond jaw 310 is removed and a top down view of first jaw surface 350of first jaw 340 is shown. In some embodiments, imaging sensor 360 islocated on first jaw 340, such as on an offset surface, flange, or areathat allows imaging sensor 360 to view, at an upward angle and/ordegree, end effector 300 of end effector 220, such as a space or areabetween first jaw surface 350 and second jaw surface 320. Thus, imagingsensor 360 may correspond to a forward looking sensor capable of imagingin front of end effector 220, an imaging sensor perpendicular to firstjaw 340 and/or second jaw 310 capable of imaging an area or spacebetween first jaw surface 350 and second jaw surface 320, and/or acombination of the two. Imaging sensor 360 is shown near distal tip ofend effector 220 so as to provide imaging of the immediate area atand/or around end effector 300 of end effector 220. Imaging sensor 360is therefore shown to be opposite second jaw 310 to allow for imaging ofan area between first jaw 340 and second jaw 310, such as a materialwithin an area between first jaw surface 350 and second jaw surface 320that is captured and/or grasped. In some embodiments, imaging sensor 360may also and/or alternatively be located on or at second jaw 310, sothat imaging sensor 360 may also and/or alternatively be opposite firstjaw 340. This allows for direct viewing of first jaw surface 350 and/ordetermination of the material in the area between first jaw surface 350and second jaw surface 320. However, it should be further understoodthat although the description below is in the context of an image sensorinstrument near the distal tip of an end effector, the aspects sodescribed may be applicable to instruments with other placements of animaging sensor along a tool and/or along an end effector.

In some embodiments of FIG. 3B, first jaw 340 is designed to receive areplaceable staple cartridge holding a plurality of staples and aplurality of staple pushers. FIG. 3B shows a top view of first jawsurface 350 having six rows of staple slots 370 through which thestaples may be applied to grasped tissue upon actuation of end effector220. The rows of staple slots 370 include three rows on each side of acutting slot 380. A cutting blade may be located somewhat proximally tothe distal ends of end effector 300 so that cutting of any graspedtissue trails the firing of the staples along both sides of the cuttingline of cutting slot 380. As a cutting blade is actuated, the cuttingblade travels along cutting slot 380 as well as a corresponding cuttingslot located in second jaw 310.

Placing the staples on both sides of cutting slot 380 allows for theapplication of the staples to both sides of a desired cutting line so asto close the tissue on both sides of the cutting line. The rows ofstaple slots 370 are also offset relative to each other to provide morecomplete tissue closure along both sides of the cutting line. Andalthough, FIG. 3B is shown with six rows of offset staple slots 370,each having six staple slots 370 of uniform size, one of ordinary skillwould understand that fewer or more rows of staple slots with fewer ormore staples, staple slots of varying size, and staple slots of varyingpatterns are possible. In some embodiments, the staple cartridgeproviding the staples is designed to be replaceable so that end effector220 is reusable by removing a first staple cartridge after one or moreof the staples are used and replacing it with a second staple cartridgehaving a new set of staples that can be used to further perform thesurgical procedure.

In some embodiments, when imaging using imaging sensor 360, one or moreimages, such as one or more of images 190, may be captured of a spaceand/or an area between first jaw surface 350 and second jaw surface 320.The area may correspond to an area that is graspable and/or grasped byfirst jaw 340 and second jaw 310 during use of end effector 220. Thus,the one or more images 190 may be used to estimate and/or determine anamount of the area that is filled with material in the workspace (e.g.,tissue of a patient when performing a grasping and/or stapling eventduring a medical use or operation). In some embodiments, imaging sensor360 may include a field of vision that allows imaging sensor 360 toimage an entire scene between first jaw surface 350 and second jawsurface 320, and/or a substantial portion of the scene. In someembodiments, imaging sensor 360 may capture the image as first jaw 340and second jaw 310 move into position to grasp a material, and maycontinue to capture and/or alternatively capture one or more imagesafter grasping the material. The images may therefore present an amountand/or presence of the material occupying the area or space betweenfirst jaw surface 350 and second jaw surface 320. Imaging sensor 360 maybe located at, along, or nearby first jaw 340 and/or second jaw 310 inorder to provide imaging of the space through one or more imagescaptured from their respective point of views.

Thus, a percentage fill of the space and/or area between first jawsurface 350 and second jaw surface 320 may be approximated based on thevisible material (e.g., tissue) between first jaw surface 350 and secondjaw surface 320. Utilizing the percentage fill of the space betweenfirst jaw surface 350 and second jaw surface 320, an amount of thematerial that is grasped and/or graspable may be determined for use inperforming the grasping, cutting, and/or stapling actions describedabove for end effector 220. The percentage fill may correspond to apercentage occupancy of the material in the space between first jawsurface 350 and second jaw surface 320, an amount or volume of thematerial, and/or another metric to measure the material that is graspedor graspable by end effector 220 (e.g., using first jaw 340 and secondjaw 310). Light may be provided by an illumination system in order toview and/or better illuminate the space between first jaw surface 350and second jaw surface 320 so that the amount and/or percentage of thespace that is filled with the material may be better determined. Imagingsensor 360 may be located near the distal tip of one or more of firstjaw 340 and/or second jaw 310 in order to provide imaging around anobject in an environment or workplace, such as around tissue whenoperating within a patient. In some embodiments, imaging sensor 360 mayalso and/or alternatively correspond to a forward-looking imaging sensorthat captures images of the material or structures prior to and/orduring entry in the area and/or space between first jaw surface 350 andsecond jaw surface 320. In some embodiments, imaging sensor 360 may alsoprovide imaging and/or analysis of normally hidden objects when locatedat the distal tip of end effector 220.

In some embodiments, imaging sensor 360 may additionally be used todetermine additional properties of the material at and/or nearby endeffector 220 using the one or more images captured by imaging sensor360, such as a material grasped and/or graspable between first jawsurface 350 and second jaw surface 320. For example, where imagingsensor 360 corresponds to and/or includes an infrared sensor, adesiccation level and/or amount of the tissue may be determined prior toand/or during grasping the material. In some embodiments, imaging sensor360 may utilize specific wavelengths of light, such as at 1600 nm, 3500nm, and/or another wavelength at which water absorbs and/or transmitslight (e.g., better than other relative wavelengths) to determine thedesiccation level of the material. For example, in certain states (e.g.,temperature, solid/liquid/gaseous phase) water may better absorb ortransmit light through the medium (e.g., the current state of the waterparticles), than at other wavelengths so as to provide better absorptionand/or transmittance of radiation through the particular medium.Radiation may be better absorbed or transmitted such that a detectorarray may determine the relative amount or level (e.g., desiccationlevel) of the water based on the amount of radiation transmitted throughthe material and/or medium. A first wavelength may therefore be betterabsorbed or transmitted through and/or by a material or medium of theliquid that is more strongly or weakly absorbed or transmitted by thatmaterial and/or medium. Thus, various states of the water may stronglyor weakly absorb and/or transmit radiance that is incident through theparticles.

Desiccation of the material (e.g., tissue) may occur prior to and/orduring a grasping action by end effector 300. When a grasping eventoccurs, liquid (e.g., water) may be desiccated from the material (e.g.,tissue). In such embodiments, by viewing the amount of light at eachpixel, a water content and/or desiccation of the material may bedetermined, and/or a state of compression of the material may beidentified by the imaging system. When utilizing more than onewavelength, such as a wavelength at which water absorbs and anotherwhere water transmits, the imaging system may utilize a ratio ofdetected radiation at each wavelength to more sensitively detect apresence of water and/or other fluid with the material. In someembodiments, imaging sensor 360 may also be used to determine thepresence of other fluids, such as blood, other bodily fluids, and/or thelike during a medical operation, which may be utilized to assist inactions by end effector 220 and diagnosis of issues within a workspace(e.g., a patient). In some embodiments, imaging sensor 360 may also beused to determine a fluorescence of a material using one or morewavelengths of light to determine material type, desiccation, and/orother property. For example, 800 nm light may be used to triggerfluorescence in tissue. When determining such properties, imaging sensor360 may provide the one or more images and/or other data to an imagingsystem, such as imaging system 260, which may process the images usingone or more modules, such as image control module 180.

As discussed above and further emphasized here, FIGS. 3A and 3B aremerely examples which should not unduly limit the scope of the claims.One of ordinary skill in the art would recognize many variations,alternatives, and modifications. According to some embodiments, arelative size of the surface areas of the one or more of first jawsurface 350, second jaw surface 320, and/or imaging sensor 360 may bedifferent than as depicted in FIGS. 3A and 3B. According to someembodiments, the relative height, placement location, orientation,and/or field of vision of imaging sensor 360 may be different than asdepicted in FIGS. 3A and 3B. In some examples, imaging sensor 360 may beflush with first jaw 340 and/or recessed below first jaw 340. In someembodiments, an axial length (from proximal to distal) along endeffector 300 of imaging sensor 360 may be longer, shorter, and/or ofdifferent lengths.

According to some embodiments, control of an end effector, such as endeffector 220, that supports both grasping (e.g., using opposing jaws)and image capture and delivery (e.g., using imaging sensor 360) aretypically controlled using separate systems. For example, a drive systemand a corresponding grasp control module may control the grasping whilean imaging system and a corresponding image control module may controlthe image capture, delivery, and/or processing. In some examples, theremay be little or no cooperation between drive system/grasp controlmodule and the imaging system/image control module. That is, the drivesystem/grasp control module may control grasping based on mechanicaland/or kinematic properties of the grasped material and not the materialand spatial occupancy properties of the grasped material that indicatewhether grasping, sealing, and/or cutting are occurring satisfactorily.Similarly, the imaging system/image control module may control imagingand image display based on material and spatial occupancy properties ofthe grasped material and not the mechanical and/or kinematic propertiesof the material that indicate whether a grasp of the material that islikely to result in good grasping, sealing, and/or cutting has beenobtained. Accordingly, better sealing and cutting of a grasped materialmay be obtained when the drive system/grasp control module and theimaging system/image control module work together to control both thegrasping and imaging so that both the grasping and imaging work tocomplement each other.

FIGS. 4A and 4B are simplified side and top views of a end effector 400of a tool according to some embodiments. In some embodiments, endeffector 400 is consistent with either or both of the jaws of endeffector 220. As shown in FIGS. 4A and 4B, end effector 400 located atthe distal end of tool 200 (e.g., end effector 220) includes a mechanismfor jaw closure, tissue stapling, and tissue cutting similar to endeffector 300 in FIGS. 3A and 3B. And although end effector 220 is shownand described with one fixed and one movable jaw, one of ordinary skillwould understand that the distal end of tool 200 could be modified touse two movable jaws. It should be further understood that although thedescription below is in the context of a grasping, stapling, and cuttinginstrument that simultaneously grasps, staples, and cuts tissue, theaspects so described may be applicable to instruments with or withoutcutting features, instruments supporting fusing rather than stapling,and/or the like.

As shown in FIG. 4A, end effector 400 (similar to end effector 300)shows a cut-way side view of end effector 220 prior to actuation so thatthe jaws of end effector 220 are shown in an open position. As shown,end effector includes a first jaw 440 that is fixed and a second jaw 410that is movable about a pivot point 430 near its proximal end. In thecontext of a stapling instrument, second jaw 410 may alternatively bereferred to as anvil 410. In the embodiments shown in FIG. 4A, first jaw440 further includes a first jaw surface 450 and second jaw 410 furtherincludes a second jaw surface 420. As discussed herein, first jaw 440,second jaw 410, and pivot point 430 function the same or similar tofirst jaw 340, second jaw 310, and pivot point 430. However, as shown inFIGS. 4A and 4B, a different imaging sensor and system may be arrangedon end effector 400 and utilized with end effector 220 to provideadditional and/or different imaging configurations and processes.

In FIG. 4A, end effector 400 further includes a first imaging module 460and a second imaging module 470 that collectively may constitute animaging sensor that may capture images of a scene at or in proximity toend effector 400 during use, movement, and/or actuation of end effector400, for example, images of a space and/or area between first jawsurface 450 and second jaw surface 420. First imaging module 460 andsecond imaging module 470 may correspond to a sensor package capable ofcapturing still images and/or video of a scene, which may be reflectiveimaging or trans-illumination imaging. In some embodiments, both firstimaging module 460 and second imaging module 470 include detector sensorarrays configured to detect incoming radiation, such as a sensor arrayhaving one or more sensor pixels arranged in array or other fashioncapable of detecting incident radiation on the sensor pixel. Thus, firstimaging module 460 and second imaging module 470 may be capable ofcapturing visual images at or nearby tool 200 using visible light, suchas the space/area between first jaw surface 450 and second jaw surface420 in an open or closed position (e.g., before and/or after grasping amaterial respectively). In such embodiments, first imaging module 460may correspond to a charge-coupled device (CCD) or complementarymetal-oxide semiconductor (CMOS) image sensor. In some embodiments,first imaging module 460 and second imaging module 470 may also orinstead include a thermal and/or infrared sensor that may be capable ofcapturing infrared images of the scene. In some embodiments, the pixelsmay capture an image at a point in time based on incoming radiationand/or may reconstruct an image over time based on movement of endeffector 220 and the incoming radiation.

First imaging module 460 and second imaging module 470 may correspond toa single pixel wide array or narrow multiple pixel wide array ofdetector sensors that may detect incoming radiation incident on firstimaging module 460 and second imaging module 470. First imaging module460 and second imaging module 470 may run a length of first jaw 440 andsecond jaw 410, respectively, and/or may only run a portion of thelength of their respective jaw. Moreover, first imaging module 460 andsecond imaging module 470 may each have separate and/or the samelengths. In some embodiments, one or more images (e.g., one or more ofimages 190) captured by first imaging module 460 and second imagingmodule 470 may correspond to separate images captured by theirrespective camera and displayed on an interface, or may be used toreconstruct an image, such as a three-dimensional reconstruction, of amaterial at and/or proximate to end effector 400 (e.g., between firstjaw surface 450 and second jaw surface 420).

In some embodiments, the sensor package provided by first imaging module460 and second imaging module 470 may provide trans-illumination imagingof radiation through a material. In such embodiments, one of firstimaging module 460 and second imaging module 470 may correspond to aradiation emitter (e.g., an infrared emitted, ultraviolet emitter,and/or emitter of another wavelength and/or radiation type), while theother one of first imaging module 460 and second imaging module 470 maycorrespond to an infrared detector or other image sensor having pixelssensitive to infrared, ultraviolet, and/or another wavelength radiation(e.g., a pair of corresponding emitter and detector sensor pixel(s)). Insuch embodiments, moisture levels in a material, absorption,desiccation, and/or other property of the material may be determinedusing first imaging module 460 and second imaging module 470, asdiscussed below. In some embodiments, one of first imaging module 460and second imaging module 470 may be a leaky fiber optic, such as afiber optic bent at such a degree that light within the fiber exceedsthe critical angle, escapes, and provides illumination, or a fiber opticthat leaks light (e.g., a Corning® Fibrance® optical fiber). The otherone of first imaging module 460 and second imaging module 470 maycorrespond to a sensing fiber, that may similarly be bent at anglessufficient to act as detectors from emitted light from the leaky fiberand/or may be a straight fiber optic similarly capable of sensing light.In such embodiments, the amount of returning light detected by thesensing fiber may be analyzed as discussed below.

Similar to imaging sensor 360, imaging using first imaging module 460and second imaging module 470 on end effector 220 is accomplished byimage control module 180 with imaging system 260 from the proximal endof end effector 220 exchanging electrical signals and/or the like withfirst imaging module 460 and second imaging module 470. Thus, one ormore components of first imaging module 460 and second imaging module470 are coupled to the one or more image delivery mechanisms 270. Itshould be further understood that although the description below is inthe context of an image sensor instrument that captures visual imagesand/or infrared images, the aspects so described may be applicable toinstruments with other types of sensors capable of capturing other typesof radiation and/or the like. In some embodiments, first imaging module460, second imaging module 470, and/or another component of end effector400 (not shown) may also include an illumination and/or lighting system,as discussed in reference to FIGS. 3A and 3B. In such embodiments, anemitter may be located with, at, or proximate to first imaging module460 and/or second imaging module 470 to provide the illumination. Insome embodiments, the illumination emitter may be located on one or moreof first jaw 440 and/or second jaw 410, or may be located elsewhere,such as on an endoscope associated with end effector 400 and locatedproximate to first imaging module 460 and/or second imaging module 470so that illumination may be provided to the scene.

In FIG. 4B a second orientation of end effector 400 is shown, where atop down view of first jaw surface 450 of first jaw 440 is shown with abottom up view of second jaw surface 420 of second jaw 410. In someembodiments, first imaging module 460 is located on or along a side offirst jaw 440, such as running at least a portion of a length of a sideof first jaw 440, and/or at an offset surface, flange, or area of firstjaw 440 that allows first imaging module 460 to view, at an upward angleor degree, end effector 400 of end effector 220, such as a space and/orarea between first jaw surface 450 and second jaw surface 420.Similarly, second imaging module 470 is located on and/or along a sideof second jaw 410, such as running at least a portion of a length of aside of second jaw 410, and/or at an offset surface, flange, and/or areaof second jaw 410 that allows second imaging module 470 to view, at adownward angle and/or degree, end effector 400 of end effector 220, suchas the space or area between first jaw surface 450 and second jawsurface 420.

Thus, first imaging module 460 and second imaging module 470 may beparallel to an edge of their respective first jaw 440 and second jaw 410that is capable of imaging at and/or around end effector 400, includingthe area and/or space between first jaw surface 450 and second jawsurface 420. First imaging module 460 is shown to be opposite second jaw410 to allow for imaging of an area between first jaw 440 and second jaw410, such as a material within an area between first jaw surface 450 andsecond jaw surface 420 that is captured and/or grasped. Similarly,second imaging module 470 is shown to be opposite first jaw 440 to allowfor imaging of the same area. In some embodiments, first imaging module460 and second imaging module 470 are located opposite as shown in FIG.4 b to allow for emission of radiation by one of first imaging module460 and second imaging module 470 and detection of radiation by theother (e.g., visible or infrared light), but other configurations may bepossible. This allows first imaging module 460 and second imaging module470 to act as a emitter/detector pair that provides detection ofmaterial properties for materials between first imaging module 460 andsecond imaging module 470, as discussed below. It should be furtherunderstood that although the description below is in the context of animage sensor instrument at the side parallel edges of an end effector,the aspects so described may be applicable to instruments with otherplacements of an imaging sensor along a tool and/or at an end effector.

Additionally in FIG. 4B, first jaw 440 is designed to receive areplaceable staple cartridge holding a plurality of staples and aplurality of staple pushers. FIG. 4B shows a top view of first jawsurface 450 having six rows of staple slots 480 through which thestaples may be applied to grasped tissue upon actuation of end effector220 as described in reference to staple slots 370 in FIG. 3B andfunctioning the same or similar. FIG. 4B further shows a cutting slot490 described in reference to cutting slot 380 in FIG. 3B andfunctioning the same and/or similarly.

In some embodiments, when imaging using first imaging module 460 andsecond imaging module 470, one or more images, such as one or more ofimages 190, may be captured of a space or an area between first jawsurface 450 and second jaw surface 420. The one or more images maycorrespond to visual images of the area or space, which may detectpresence of material and other visible characteristics within the areaby an imaging system (e.g., image control module 180 and/or imagingsystem 260). The area may correspond to an area that is graspable and/orgrasped by first jaw 440 and second jaw 410 during use of end effector220. Thus, the one or more images may be used to estimate and/ordetermine an amount of the area that is filled with material in theworkspace (e.g., tissue of a patient when performing a grasping and/orstapling event during a medical use or operation). A percentage fill ofthe space or area between first jaw surface 450 and second jaw surface420 may be approximated based on the visible material (e.g., tissue)between first jaw surface 450 and second jaw surface 420, such as apercentage occupancy of the material in the space between first jawsurface 450 and second jaw surface 420, similar to FIGS. 3A and 3B.Light may similarly be provided by an illumination system in conjunctionwith the imaging system.

In some embodiments, other types of images or image data may becaptured, such as infrared images and/or data on transmission propertiesof material in the area and/or space between first imaging module 460and second imaging module 470 (e.g., between first jaw surface 450 andsecond jaw surface 420). For example, trans-illumination infraredimaging may be utilized by first imaging module 460 and second imagingmodule 470 (e.g., through an emitter/detector pair formed by firstimaging module 460 and second imaging module 470) to detect presence ofwater in tissue based on absorption of infrared light at certainwavelengths by an imaging system. The presence and/or amount of moisturedetected by the imaging system between first jaw surface 450 and secondjaw surface 420 may be utilized to determine the percentage fill of endeffector 400 with material. In some embodiments, as the material isgrasped and squeezed between first jaw surface 450 and second jawsurface 420 by end effector 400 during actuation of end effector 220,moisture (e.g., water) may be squeezed out of the material (e.g.,tissue) causing desiccation. The presence and/or amount of moisture inthe material may further be utilized by the imaging system to determinethe percentage fill of the space or area with the material during thegrasping action by end effector 220. The desiccation level during agrasping event determined through the one or more images may be used bythe imaging system to determine a state of compression of the material,such as an amount of pressure applied and/or amount of compressedmaterial.

In some embodiments, first imaging module 460 and second imaging module470 may utilize specific wavelengths of light, such as at 1600 nm, 3500nm, and/or another wavelength at which water absorbs and/or transmitslight to determine the desiccation level of the material (e.g., thewater content of the material). In such embodiments, by viewing theamount of light scene at each pixel, a water content and/or desiccationof the material may be determined. When utilizing more than onewavelength, such as a wavelength at which water absorbs and anotherwavelength where water transmits, the imaging system may utilize a ratioof detected radiation at each wavelength to more sensitively detect apresence of water or other fluid with a material. In some embodiments,first imaging module 460 may also be used to determine the presence ofother fluids, such as blood or stomach fluid during a medical operation,which may be utilized to assist in actions by end effector 220 anddiagnosis of issues within a workspace (e.g., a patient). In someembodiments, first imaging module 460 and second imaging module 470 mayalso be used to determine a fluorescence of a material using one or morewavelengths of light to determine material type, desiccation, and/orother property (e.g., 800 nm as in FIGS. 3A and 3B). Similar to FIGS. 3Aand 3B, an imaging system (e.g., imaging system 260) may process theimages using one or more modules (e.g., image control module 180) inorder to determine the aforementioned properties.

In some embodiments, transmissivity (e.g., transmission level ofradiation through a substance) of visible light through an area or spacemay also be utilized to detect presence of material between firstimaging module 460 and second imaging module 470, as well as an amountand/or percentage fill of the area between first jaw surface 450 andsecond jaw surface 420. For example, when first imaging module 460 andsecond imaging module 470 correspond to an emitter/detector pairutilizing leaky fiber optics or other visible light pairs, the amount ofdetected and/or returning light in the detector of the emitter/detectorpair may be used to determine properties of the material, includingstate of compression or pressure applied to the material, desiccationlevel or amount during an action by end effector 220, and the like. Theimaging system may determine the transmission properties of the materialutilizing such an imaging sensor pair, which may be analyzed foradditional properties of the material by the imaging system.

As discussed above and further emphasized here, FIGS. 4A and 4B aremerely examples which should not unduly limit the scope of the claims.One of ordinary skill in the art would recognize many variations,alternatives, and modifications. According to some embodiments, arelative size of the surface areas of the one or more of first jawsurface 450, second jaw surface 420, first imaging module 460, and/orsecond imaging module 470 may be different than as depicted in FIGS. 4Aand 4B. According to some embodiments, the relative height, placementlocation, orientation, and/or field of vision of first imaging module460 and/or second imaging module 470 may be different than as depictedin FIGS. 4A and 4B. In some examples, first imaging module 460 and/orsecond imaging module 470 may be flush with their respective one offirst jaw 440 and second jaw 410 and/or recessed below their respectivejaw. In some embodiments, an axial length (from proximal to distal)along end effector 400 of each of first imaging module 460 and/or secondimaging module 470 may be longer, shorter, and/or of different lengths.Additionally, when discussed as an emitter/detector pair, it isunderstood that first linear imaging sensor 406 and second imagingmodule 470 may each function as the emitter and/or detector (e.g., areinterchangeable depending on system requirements and functionality).

According to some embodiments, control of an end effector, such as endeffector 220, that supports both grasping (e.g., using opposing jaws)and image capture and delivery (e.g., using first imaging module 460)are typically controlled using separate systems. For example, a drivesystem and a corresponding grasp control module may control the graspingwhile an imaging system and a corresponding image control module maycontrol the image capture, delivery, and processing. In some examples,there may be little or no cooperation between drive system/grasp controlmodule and the imaging system/image control module. That is, the drivesystem/grasp control module may control grasping based on mechanicaland/or kinematic properties of the grasped material and not the materialand spatial occupancy properties of the grasped material that indicatewhether grasping, sealing, and/or cutting are occurring satisfactorily.Similarly, the imaging system/image control module may control imagingand image display based on material and spatial occupancy properties ofthe grasped material and not the mechanical and/or kinematic propertiesof the material that indicate whether a grasp of the material that islikely to result in good grasping, sealing, and/or cutting has beenobtained. Accordingly, better sealing and cutting of a grasped materialmay be obtained when the drive system/grasp control module and theimaging system/image control module work together to control both thegrasping and imaging so that both the grasping and imaging work tocomplement each other.

FIG. 5 is a simplified diagram of a method 500 for grasping and energydelivery according to some embodiments. One or more of the processes510-560 of method 500 may be implemented, at least in part, in the formof executable code stored on non-transitory, tangible, machine readablemedia that when run by one or more processors (e.g., the processor 150in control unit 140) may cause the one or more processors to perform oneor more of the processes 510-560. In some embodiments, method 500 may beperformed by one or more modules, such as grasp control module 170and/or image control module 180. In some embodiments, portions of method500 associated with grasping (e.g., sensing of mechanical and/orkinematic information and mechanical control of grasping jaws) may beperformed by grasp control module 170 and portions of method 500associated with image capture and analysis (e.g., capturing, recording,and analyzing images for determination of material presence andproperties) may be performed by image control module 180 with graspcontrol module 170 and image control module 180 cooperating to sharesensor and control information so as to optimize image capture with agrasped or graspable material. In some embodiments, process 560 isoptional and may be omitted, or may be performed prior to one or more ofprocesses 510, 520, 530, 540, and/or 550. In some embodiments, method500 may be performed in a different order than the order implied by FIG.5 . In some examples, processes 520, 530, 540, 550, and 560 may beperformed before process 510 and/or processes 520, 530, 540, 550, and560 may be performed concurrently with process 510. In some examples,process 520 is optional and may be omitted as necessary. In someembodiments, both processes 520 and 530 may occur concurrently.

At a process 510, a material is grasped. In some examples, the materialmay be grasped between the jaws of an end effector, such as end effector220. In some examples, the end effector may be consistent with endeffector 300 and/or end effector 400. In some examples, the material maybe grasped using a drive system, such as drive system 240, under thecontrol of a grasp control system, such as grasp control module 170. Insome examples, the grasp may occur based on a command received from anoperator. In some examples, the grasp may include actuation of the jawsuntil a desired angle between the jaws is reached, a desired separationbetween the jaws is reached, and/or a desired force or torque limitindicating a desired grasp strength is reached. In some examples, thegrasp may actuate the jaws to a desired position set point (e.g., adesired angle and/or separation between the jaws) subject to an upperforce and/or torque limit. In some examples, the force or torque limitsmay be implemented as a current limit on the one or more actuators usedto actuate the jaws.

At a process 520, illumination is emitted. Illumination may be providedto a scene at or nearby the distal end of tool 200 where end effector220 is currently operating within a workspace. In some examples, endeffector 220 is part of a medical tool utilized inside a patient, andmay correspond to a grasping, sealing (e.g., stapling), and/or cuttingtool with the material grasped at process 510. Thus, illumination may beprovided to the tissue of a patient that is proximate to end effector220. Illumination may be provided by an illumination system thatincludes a LED, other diode, light pipe, fiber optic, and/or lightemission device at or nearby the distal end of tool 200 and nearby endeffector 220 (e.g., located at and/or nearby imaging sensor 360 or firstand second linear imaging sensors 460/470). The illumination may consistof visible light, and/or may be other radiation including infraredradiation. In some embodiments, illumination may be provided by anotheror an additional articulating arm or tool, for example, provided adifferent end effector.

In some embodiments, illumination may be provided by one or more opticalfibers that span a distance from a light source and illumination controlsystem at the proximal end of tool 200 to the illuminating device at thedistal end of the tool and nearby end effector 220. The optical fibersmay be combined with drive mechanisms 250 and/or image deliverymechanisms 270 through an arm and other components of tool 200. Thelight or illumination source at the proximal end of tool 200 maycorrespond to a light engine having red, green, and blue LEDs and/orlaser diodes that may mix to create colored/white light, a white LED, aninfrared light source, an ultraviolet light source, and/or the like. Insome embodiments, the illumination control system may be controlled byone or more of grasp control module 170 and image control module 180.

In some embodiments, the one or more of the illumination control system,grasp control module 170, and image control module 180 may also controlambient increases in temperature caused by providing light to the scene.This may prevent overheating of sensitive material caused by theillumination. Thus, end effector 220 and/or another component of tool200 may also include a temperature measurement component (e.g.,thermometer, thermocouples, thermal resistors, thermistor, etc.) locatedon and/or near one or both of the jaws that may measure temperature atand/or near the scene and provide feedback of illumination correction.In some examples, the temperature of the grasped material may bedetermined using an infrared sensor, such as an imaging sensor that mayfurther be used to capture one or more images discussed below. The oneor more systems may then adjust illumination and/or turn offillumination as necessary. However, in other embodiments, temperaturechange by the illumination may be unimportant (e.g., during heatingevents by end effector 220).

At a process 530, one or more images are received. Image capture andreceipt may be performed by an imaging system, such as imaging system260, which may be controlled by an image control module, such as imagecontrol module 180. In some examples, imaging and receipt of the one ormore images may occur based on a command received from an operator,and/or may occur automatically based on an interaction with drive system240 under the control of grasp control module 170 when grasping amaterial in a workspace.

In some embodiments as shown in FIGS. 3A and 3B, the one or more imagesmay be received from a single imaging sensor, such as imaging sensor360, located at or near a distal tip of tool 200, such as near a distaltip of end effector 300 (e.g., either the tip of first jaw 340 or secondjaw 310). In such embodiments, the one or more images may be capturedfrom a single sensor and transmitted to imaging system 260 and/or imagecontrol module 180 from imaging sensor 360. In other embodiments asshown in FIGS. 4A and 4B, the one or more images may be received from animaging module, such as first imaging module 460 and second imagingmodule 470 that are located in opposing positions along end effector400, such as at the sides of first jaw 440 and second jaw 410,respectively. In such embodiments, the one or more images may correspondto a plurality of images and/or one or more combined images using thetwo or more imaging modules.

In some embodiments, the one or more images may correspond to visualimages captured using visible light. Such images may therefore include avisual representation of a scene, which may include objects in thescene, such as a material within a workspace. Thus, the one or moreimages may correspond to visual data shown in a visual representation ofa scene using visual light (e.g., through reflective imaging anddetection of visual light by one or more imaging sensors).

In some embodiments, the one or more images may include infrared imagedata, transmission data of light and/or other radiation through amaterial, and the like. In such embodiments, the images may thereforeinclude additional information, such as amount of visible/infrared lightabsorbed or transmitted through a material or liquid associated with thematerial and/or absorption of the light emitted from the emitter anddetected by the detector (e.g., an amount or percentage of the emittedlight that is detected by the detector). The one or more images mayinclude data of emitted light wavelength(s) for use during processingand transmission/absorption data of the emitted light. Thus, the one ormore images may include image data detected of trans-illumination, scenefluorescence, and/or the like for the material located in the scene.

At a process 540, one or more properties of the material captured in theone or more images are determined. The one or more images may correspondto image data of a scene, which includes data of the material capturedusing one or more imaging sensors located on and/or near end effector220. As discussed above, the image data may correspond to visual imagedata, such as detected visual light that is radiant, transmissive (e.g.,during trans-illumination), fluorescent, and/or reflective from and/orthrough a material in a scene, and/or may correspond to detection ofother types of radiation in a scene. In addition to the below propertiesdetermined from one or more images, changes to the properties may alsobe determined, for example, when utilizing a plurality of images and/orvideo.

In some examples, the one or more properties may be determined from oneor more images obtained from an imaging sensor of the jaws and thegrasped or graspable material. Thus, the one or more properties may beused to determine a presence and/or absence of a material in a scene aswell as the material within an area and/or space between thearticulating jaws of end effector 220 (e.g., first jaw 340/440 andsecond jaw 310/410). In some embodiments, the one or more images mayalso be used to determine a length and/or amount of fill of thearea/space between the jaws, for example, by detecting a length, width,and/or height of the material entering and occupying the jaws.

In some examples, the one or more properties may include a desiccationlevel (e.g., moisture content) of the grasped and/or graspable material.In some examples, the desiccation level may provide an indicator of alevel of current material sealing, an indication of whether the materialis ready for cutting and/or sealing (e.g., it may be advantageous tosqueeze moisture out the material by grasping before cutting and/orsealing). In some examples, the desiccation level may be determined fromthe moisture and/or water levels prior to and/or after grasping andclamping by end effector 220. In some examples, the desiccation levelmay be determined based on the presence and amount of water in thetissue prior to and/or after grasping the material utilizing light. Forexample, at certain wavelengths (e.g., approximately 1600 nm and/or 3500nm), light is significantly absorbed by water. Thus, utilizing light atthis wavelength with the material allows a determination of watercontent within a material. In some embodiments, multiple lightwavelengths, such as a wavelength at which water absorbs and anotherwavelength at which it water transmits the light, may be utilized todetect the presence and amount of water with more sensitivity and/oraccuracy. In some embodiments, light at 800 nm may cause fluorescence ofliquid (e.g., a fluorescing compound, such as indocyanine green and/orthe like) from a material to determine the material's desiccation level.

In some examples, the state of compression, stiffness, and/or pressureapplied to the grasped material may be determined using the one or moreimages, as well as other determined properties. In some examples, thedesiccation level may be used to determine the state of compression. Insome examples, the stiffness and/or state of compression of the graspedmaterial may be determined from the jaw angle and/or separation. In someexamples, a type of fluid, fluorescence of the fluid and/or material,trans-illumination of light through the fluid or material, and/orabsorption of light by the fluid or material may be determined using theone or more images.

In some embodiments, at process 540, additional data may be determined,such as a percentage fill of the jaws of end effector 220 (e.g., anamount or percentage that the space/area between first jaw 340/440 andsecond jaw 310/410 is filled prior to grasping a material and/or aftergrasping a material). In some embodiments, the percentage of thearea/space filled with the material may be determined from visual imagedata, such as the presence and length/amount of material between thejaws. In some embodiments, the percentage (or other measurement) may bedetermined based on desiccation level information for the material priorto grasping or when grasped by the jaws. For example, when applyingpressure to a material, liquid (e.g., water) may be squeezed out of thematerial. Based on the amount of water desiccated from the material(e.g., before and/or after grasping), the percentage of the area/spacemay be determined.

At a process 550, the one or more properties are displayed. The one ormore properties may be displayed on an interface of a device or machineassociated with tool 200, such as one in connection with control unit140 that may display data used for operation of device 110 and tool 200within a workspace. The one or more properties may be displayed with theone or more images, and/or may be displayed independent of the one ormore images. Additionally, the one or more properties may be displayedwith additional data that caused the one or more properties, such as aclamping pressure and/or state of compression of a material, adesiccation level of the material, and/or other data determined from theone or more images during process 540. Where the one or more propertiesinclude specific data for portions, areas, and/or points within theworkspace and/or on the material, the properties may be displayed withthose locations and/or used to identify those locations. In someembodiments, the one or more properties may also be displayed withinstructions to operate and/or move tool 200 and/or end effector 220 inorder to properly operate tool 200 and/or end effector 220 within theworkspace. For example, where the one or more properties correspond to apercentage fill of a space and/or area between two jaws on end effector220, the one or more properties may be displayed with instructions tograsp more or less of the material (e.g., movement directions, graspingand/or release directions, etc.). The data can also be used to controlthe end effector, e.g. adjusting the clamping force based on percentagefill. In some embodiments, feedback from one or more sensors on endeffector 220 may be used within a closed loop control of end effector220. The feedback may be used to adjust a property or usage of endeffector 220, such as by adjusting pressure or force imparted by endeffector 220 onto the material within or graspable by end effector 220.

In some embodiments, display of the one or more images with the one ormore properties may occur in a picture-in-picture, separate view pane,off-to-side view, and/or over the jaws and/or other part of end effector220 in order to provide context to the images. Moreover, at process 550,in some embodiments, video stabilization may be applied during captureof video data. The video stabilization may fix particular points on thematerial to locations within an interface to provide a stabilized frameof reference. In some embodiments, additional types of videostabilization may be applied.

At an optional process 560, one or more orientation hints are provided.In some embodiments, the one or more orientation hints may be determinedrelative to end effector 220 in a workspace so that an orientation ofthe one or more images captured during process 530 relative to endeffector 220 and/or the jaws of end effector 220 that may be captured bya separate imaging device, such as an endoscope used to capture imagesof end effector 220 and/or a workspace around end effector 220, therebydisplaying an orientation of the one or more images captured duringprocess 530. In some embodiments, the orientation hints may correspondto an image, graphic, and/or animation within images of the workspace,such as a T-pose of a body in a prone position with arms of the bodythat are perpendicular to a horizontal axis of the body (e.g., the chestand legs) with the face of the body showing the direction of view and/orview up for the images captured during process 530. In some embodiments,the orientation hints may also show movement of the end effector withinthe workspace.

In some embodiments, the orientation hints may be animated to display a“fly-in” or movement of end effector 220 as it enters and moves withinthe workspace along its various DOFs. As such, the orientation hints maybe determined based on inputs and/or automatic movements of tool 200 asit enters and is used within the workspace. In other embodiments, theorientation hints may be fixed to show a current position and/ororientation of the one or more images captured during process 530 and/ormay be animated to show how the position and/or orientation of the oneor more images may have been captured over time. In some embodiments,when providing the one or more orientation hints, an animation of theone or more orientation hints may be utilized prior to, with, and/orafter display of the one or more images, which may occur through theinterface. In some embodiments, the animation of the one or moreorientation hints (e.g., T-pose of a prone body) may start in an upperleft corner of the interface and proceed to the lower right corner ofimages showing the workspace and then displaying the one or more imagesfor proper orientation of the one or more images within the workspacerelative to the view of the workspace as shown in the images captured bythe endoscope.

In some embodiments, the orientation hints may provide hints for up,down, left, right, one or more cardinal directions, or other directionalhint within the workspace, which may be determined based oninputs/movements of tool 200 as well as other sensors (e.g., a gravitysensor, compass, etc.). The one or more orientation hints may be furtherdetermined and output as described below.

FIGS. 6A, 6B, and 6C are simplified orientation view hints for aninterface of a computer-assisted system of FIG. 1 having an end effectorof FIGS. 3A and 3B or FIGS. 4A and 4B according to some embodiments.FIGS. 6A-C display an exemplary interface display when displayingorientation hints during use of a tool within a workspace, such as theorientation hints discussed in reference to process 560 of method 500.In some embodiments, FIGS. 6A-6C show images of an end effector 610,such as might be captured by an imaging device located in the workspace(e.g., an endoscope mounted to another repositionable arm). In someexamples, end effector 610 may correspond to end effector 220, endeffector 300, and/or end effector 400 that is utilized within aworkspace, and may include an imaging sensor 620 that may correspond toimaging sensor 360, first imaging module 460, and/or second imagingmodule 470 as described in further detail above. However, the visualrepresentation of end effector 610 and/or imaging sensor 620 within aninterface may also be different in some embodiments. In some examples,imaging sensor 620 may be consistent with imaging sensor 360, firstimaging module 460, and/or second imaging module 470.

As shown in FIG. 6A, an orientation hint 630 is provided in the form ofa still T-pose of a prone body. The position and/or orientation oforientation hint 630 provides hints to the operator as to theorientation of the one or more images captured by imaging sensor 620 inrelation to the view of end effector 610 being displayed to the operatoras part of FIG. 6A. Thus, orientation hint 630 provides context to theoperator regarding the direction of view and/or the view up directionfor the one or more images captured by imaging sensor 620.

In some examples, end effector 610 and/or imaging sensor 620 may behidden from view of the operator during the operation, such as bymaterial and/or tissue grasped by end effector 610. In order to providecontext and orientation of a view captured by imaging sensor 620 whenlocated on end effector 610, a direction of the view and/or of view upof imaging sensor 620 relative to the workspace in the images capturedby the endoscope may be displayed using orientation hint 630 on aninterface. The direction of view up on orientation hint 630 may beprovided by rotating and/or orienting the head of the T-pose figure inorientation hint 630 so that the face is directed in the direction ofview and the top of the head is towards view up. Thus, orientation hint630 when providing a view up orientation may be shown based on thefacial direction of the T-pose body relative to the view of end effector610 provided in the view corresponding to FIG. 6A. In some embodiments,the one or more images captured by imaging sensor 620 may be provided asa picture-in-picture image next to the images of end effector 610,imaging sensor 620, and/or orientation hint 630.

As shown in FIG. 6B, an animation of orientation hints 641, 642, and 643may be used to provide a visual representation of the view directionand/or view up orientation of view of imaging sensor 620 relative to theworkspace, as discussed above. In FIG. 6B, the animation may move anorientation hint from locations corresponding to orientation hint 641,642, and 643. In some examples, the animation may move the orientationhints from top left corner to bottom right corner of the interface,based on previous motion of end effector 610 within the workspace,and/or the like. In some examples, orientation hints 641-643 may bedifferent or separate so as to show an animation of different workspaceposes and/or directions.

As shown in FIG. 6C, an orientation hint 650 may also display one ormore relative directions of view up or another view, or may insteadutilize cardinal directions and the like, to provide an orientation hintfor a current direction of imaging sensor 620 and/or a view up directionof imaging sensor 620. For example, orientation hint 650 displays thedirection of view as a “view” arrow and a view up direction as an “up”arrow. In some examples, orientation hint 650 may be animated similar tothe T-pose orientation hint as shown in FIG. 6B.

FIGS. 7A and 7B are simplified diagrams of the distal end of aninstrument grasping a material according to some embodiments. In someexamples, FIGS. 7A and 7B depict images 700 and 770 shown asinformational for the description of FIGS. 7A and 7B. Thus, FIGS. 7A and7B present an exemplary scenario meant for the discussion of apercentage fill of jaws 730, which may be captured with otherperspectives and data by imaging sensor 620, imaging sensor 360, firstimaging module 460, and/or second imaging module 470. In this regard,images 700 and 770 may correspond to descriptive images of an endeffector 710, such as end effector 220, where an imaging sensor maycapture other images of a material graspable by the end effector, andthe side-view of end effector 710 during operation as shown in FIGS. 7Aand 7B are exemplary to discuss operation of end effector 710 with animaging sensor to determine a percentage fill of end effector 710 with amaterial. As shown in FIG. 7A, a material 720, such as tissue, may begrasped by jaws 730 toward the distal end of the jaws 730. As furthershown in FIG. 7A, a grasping zone 750 at a distal end of jaw 730 mayindicate where jaws 730 and the material 720 are in contact.Additionally, a length L1 indicates a distance between a proximal end740 of jaws 730 and a proximal end of the grasping zone 750.

Similarly, as shown in image 770 of FIG. 7B, a material 780, which maybe similar to the material 720, may be grasped by jaws 730. As furthershown in FIG. 7B, a grasping zone 750 may indicate where jaws 730 andthe material 780 are in contact. Again, a length L1 indicates a distancebetween the proximal end 740 of jaws 730 and a proximal end of thegrasping zone 750. Additionally, a length L3 indicates a distancebetween a distal end of the grasping zone 750 and a distal end of jaw730. As shown in FIG. 7B, the grasping zone 750 is in between the lengthL1 and L3. Also, the length L1 in FIG. 7A may be greater than the lengthL1 in FIG. 7B and the angle A between the jaws 730 in FIG. 7A may besmaller than the angle A between jaws 730 in FIG. 7B even when thematerial 720 and 780 have substantially the same size and/or shape. Insome embodiments, imaging sensor 360 and/or first imaging module 460with second imaging module 470 may operate to capture L1, L2, and/or L3by determining an amount of material 720 that enters grasping zone 750and/or an amount of material 720 and/or 780 within grasping zone 750. Insome examples, imaging sensor 360 may monitor, over time, an amount of amaterial as it enters jaws 730 and in combination with known informationabout motion of end effector 220 may be used to determine grasping zone750 to determine L1, L2, and/or L3, or may image part or all of graspingzone 750 having material 720 and/or or 780 within grasping zone 750 forthe determination, such as with reflective light and/or radiation. Insome embodiments, first imaging module 460 with second imaging module470 may similarly image the area using transmissive light throughmaterial 720 and/or 780 and determine grasping zone 750 by detectingwhere light and/or radiation emitted from one of first imaging module460 and second imaging module 470 is received by the other one of themodules and/or where light and/or radiation emitted by first imagingmodule 460 and/or second imaging module 470 is reflected back to firstimaging module 460 and/or second imaging module 470, respectively. Insome embodiments, the percentage fill may also be determined by imagingsensor 360 and/or first imaging module 460 with second imaging module470 through material characteristics, such as a desiccation level,transmissivity of radiation through material 720 and other space ofgrasping zone 750, fluorescence of material 720 and/or 780, and/orabsorption of radiation by material 720 and/or 780.

In some embodiments, a percentage fill of jaws 730 is determined basedon one or more of the length L1, L2, L3, and/or the like. In someembodiments, a percentage fill P1 of material 720 between jaws 730 inFIG. 7A may be determined as a function of L2/(L1+L2). Thus, in FIG. 7A,P1=L2/(L1+L2). In some embodiments, the percentage fill P2 of jaws 730in FIG. 7B with material 720 may be determined as a function ofL2/(L1+L2+L3) to include the region L3 that does not include graspablematerial after material 780 is placed further within jaws 730. Thus, inFIG. 7B, P2=L2/(L1+L2+L3). In some embodiments, the functions P1 and P2additionally depend on “other parameters” such as a procedure to beperformed on the material, a material type, other material properties(e.g., desiccation, etc.), an operator preference, a position and/ororientation of the instrument or end effector 710, and/or the like andtherefore a number of lookup tables are composed based on the otherparameters and are stored in memory.

Some examples of control units, such as control unit 140 may includenon-transitory, tangible, machine readable media that include executablecode that when run by one or more processors (e.g., processor 150) maycause the one or more processors to perform the processes of method 500.Some common forms of machine readable media that may include theprocesses of method 500 are, for example, floppy disk, flexible disk,hard disk, magnetic tape, any other magnetic medium, CD-ROM, any otheroptical medium, punch cards, paper tape, any other physical medium withpatterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chipor cartridge, and/or any other medium from which a processor or computeris adapted to read.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

What is claimed is:
 1. A computer-assisted device comprising: an endeffector having a first jaw and a second jaw; an imaging sensor mountedto the first jaw or the second jaw and configured to capture one or moreimages of a material graspable by the first jaw and the second jaw; andone or more processors coupled to the end effector and the imagingsensor, the one or more processors being configured to: receive the oneor more images from the imaging sensor; analyze the one or more imagesto determine a length of a space between the first jaw and the secondjaw filled with the material; and display, on a user interface,information describing the length of the space between the first jaw andthe second jaw filled with the material.
 2. The computer-assisted deviceof claim 1, wherein the end effector further comprises one or more of: astapling mechanism configured to staple the material; or a cuttingmechanism configured to cut the material.
 3. The computer-assisteddevice of claim 1, wherein the imaging sensor is mounted to a side ofthe first jaw or a side of the second jaw.
 4. The computer-assisteddevice of claim 1, wherein the one or more processors are furtherconfigured to display, on the user interface, the one or more images ofthe material.
 5. The computer-assisted device of claim 1, wherein theone or more processors are further configured to: analyze the one ormore images to determine at least one of a width of the material or aheight of the material; and display, on the user interface, at least oneof the width of the material or the height of the material.
 6. Thecomputer-assisted device of claim 1, wherein the one or more processorsare further configured to: analyze the one or more images to determine apercentage of the space between the first jaw and the second jaw filledby the material; and display, on the user interface, the percentage ofthe space between the first jaw and the second jaw filled by thematerial.
 7. The computer-assisted device of claim 6, wherein the one ormore processors are configured to determine the percentage of the spacebetween the first jaw and the second jaw filled by the material using afunction based on at least one of a procedure to be performed on thematerial, a type of the material, an operator preference, a position ofthe end effector, or an orientation of the end effector.
 8. Thecomputer-assisted device of claim 1, wherein the one or more processorsare further configured to analyze the one or more images to determine agrasping zone indicating where the material is in contact with at leastone of the first jaw or the second jaw.
 9. The computer-assisted deviceof claim 8, wherein to determine the grasping zone, the one or moreprocessors are configured to: determine a first distance between thematerial and a proximal end of the first jaw or the second jaw; anddetermine a second distance between the material and a distal end of thefirst jaw or the second jaw.
 10. The computer-assisted device of claim1, wherein the one or more processors are further configured to analyzethe one or more images to determine at least one of: a desiccation ofthe material; a change in the desiccation of the material; a pressureapplied to the material by the first jaw and the second jaw detected inthe one or more images; a change in the pressure applied to thematerial; a transmissivity of the material captured in the one or moreimages; a change in the transmissivity of the material; an absorption ofa liquid by the material; a change in the absorption of the liquid bythe material; a fluorescence of the material captured in the one or moreimages; or a change in the fluorescence of the material.
 11. A methodcomprising: operating, by one or more processors, an end effector havinga first jaw and a second jaw; capturing, by the one or more processorsusing an image sensor mounted to the first jaw or the second jaw, one ormore images of a material grasped by the first jaw and the second jaw;analyzing, by the one or more processors, the one or more images todetermine a length of a space between the first jaw and the second jawfilled with the material; and displaying, by the one or more processorson a user interface, information describing the length of the spacebetween the first jaw and the second jaw filled with the material. 12.The method of claim 11, further comprising displaying, by the one ormore processors on the user interface, the one or more images of thematerial.
 13. The method of claim 11, further comprising: analyzing, bythe one or more processors, the one or more images to determine at leastone of a width of the material or a height of the material; anddisplaying, by the one or more processors on the user interface, atleast one of the width of the material or a height of the material. 14.The method of claim 11, further comprising: analyzing, by the one ormore processors, the one or more images to determine a percentage of thespace between the first jaw and the second jaw filled by the material;and displaying, by the one or more processors on the user interface, thepercentage of the space between the first jaw and the second jaw filledby the material.
 15. The method of claim 11, further comprisinganalyzing, by the one or more processors, the one or more images todetermine a grasping zone indicating where the material is in contactwith at least one of the first jaw or the second jaw.
 16. The method ofclaim 15, wherein determining the grasping zone comprises: determining afirst distance between the material and a proximal end of the first jawor the second jaw; and determining a second distance between thematerial and a distal end of the first jaw or the second jaw.
 17. Anon-transitory machine-readable medium comprising a plurality ofmachine-readable instructions which when executed by one or moreprocessors are adapted to cause the one or more processors to perform amethod comprising: operating an end effector having a first jaw and asecond jaw; capturing, using an image sensor mounted to the first jaw orthe second jaw, one or more images of a material grasped by the firstjaw and the second jaw; analyzing the one or more images to determine alength of a space between the first jaw and the second jaw filled withthe material; and displaying, on a user interface, informationdescribing the length of the space between the first jaw and the secondjaw filled with the material.
 18. The non-transitory machine-readablemedium of claim 17, wherein the method further comprises displaying, onthe user interface, the one or more images of the material.
 19. Thenon-transitory machine-readable medium of claim 17, wherein the methodfurther comprises: analyzing the one or more images to determine apercentage of the space between the first jaw and the second jaw filledby the material; and displaying, on the user interface, the percentageof the space between the first jaw and the second jaw filled by thematerial.
 20. The non-transitory machine-readable medium of claim 17,wherein the method further comprises analyzing the one or more images todetermine a grasping zone indicating where the material is in contactwith at least one of the first jaw or the second jaw.